Survey Data Analysis with Kobocruncher

class: center, middle, inverse, title-slide

.title[
# Survey Data Analysis with Kobocruncher
]
.subtitle[
## Session 6 - Cleaning and Indicator Calculation
]
.author[
### <a href="https://edouard-legoupil.github.io/kobocruncher/">Link to Documentation</a> – <a href="05-Searching_Asssociation.html">Link to Previous Session</a> – <a href="07-Anonymising.html">Link to Next Session</a>
]
.date[
### Training Content as of 30 November 2022
]

---

## When do yo need to clean the data?

Survey data cleaning involves identifying and removing responses from individuals who either don’t match your target audience criteria or didn’t answer your questions thoughtfully. This filtering is done to avoid drawing misleading conclusions.

Data cleaning remains a last resort option that can be at first minimized by:

* .large[__Quality of questionnaire design__] not only to minimize social desirability and biased questions but also to ensure that the interview duration is limited (_ideally less than 45 minutes for a face to face interview and less than 25 minutes for a telephone interview_)

* .large[__Good form encoding__] - with well defined [constraints](https://xlsform.org/en/#constraints) and [skip logic](https://xlsform.org/en/#relevant) and [requirement](https://xlsform.org/en/#required) to avoid Inconsistent Responses, sufficient testing to ensure that the questions are well understood and the responses options are covering well the options
 
 * .large[__Good training for the data enumerator__] and detailed [question hints](https://xlsform.org/en/#hints) so the enumerators fills correctly the questionnaire
 
 * .large[__Sufficient data collection quality monitoring__] to identify, prevent and cure issues early on. This can be done through [High Frequency Check](https://unhcr.github.io/HighFrequencyChecks/docs/). This should help to  flag Straightlining / Patterned Responses when an enumerator is using the same answer option ("B") over and over (for instance  for at least five rows in a grid)...  
 
 
 .bg-blue[
For data quality, prevention is lot more effective, quicker and cheaper than curing. Take the time to thoroughly test the questionnaire before starting full on data collection.
]

???

---

## Situation when you will still need minimum cleaning a priori

Whatever is quality of form design, enumerator training and data collection quality monitoring, there will be still cases where cleaning will involve removing entire records:

* Remove from the dataset records where no consent were obtained and/or more broadly one a specific filter/condition (where the respondent do not meet certain criteria or data from an unreliable enumerator identified during data collection quality monitoring...
 
 * Remove duplicate respondent ID based on the original sample list
 
 * Remove from the dataset records before or at specific dates
 
 * Remove from the dataset  records when interview duration appears as outliers, either too long or too short, aka "speed responses" 
 
 
---

## How to set up filters on the data
 
---

## Situation when you will still need minimum cleaning a posteriori
 
In other cases, cleaning will involve recoding some variables 
 
 
 1. Recode un-explainable .large[__outliers for numerical questions__]. An example of this would be if you asked how much water one person use in a day and someone answered that they use 1000 liters, while the second largest usage reported is 150 liters.
 
 
 2. Recode questions consecutive from  .large[__"or other" choices__].
 
 3. Recode some questions answer as .large[__new calculated variables__] to have more balanced response categories based on frequency or potential closes meaning
 
 
 
 
---

## Outliers for numerical questions

---

## "or other" choices
---

## New calculated variables

---
class: inverse, center, middle

# TIME TO PRACTISE ON YOUR OWN!

- Open again locally and set up the outliers treatment, clean the _"or_other"_ and add calculated variables
 
 - upload and build up you data dictionary `kobo_dico()`
 
 - download again the xlsfrom and fill in the various tab for cleaning log
 
 - add the following indicator:

Do not hesitate to raise your questions in the [ticket system](https://github.com/Edouard-Legoupil/kobocruncher/issues/new) or in the chat so the training content can be improved accordingly! 
 
---
class: inverse, center, middle

### .large[.white[<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M256 512C114.6 512 0 397.4 0 256S114.6 0 256 0S512 114.6 512 256s-114.6 256-256 256zM232 120V256c0 8 4 15.5 10.7 20l96 64c11 7.4 25.9 4.4 33.3-6.7s4.4-25.9-6.7-33.3L280 243.2V120c0-13.3-10.7-24-24-24s-24 10.7-24 24z"/></svg>] **Let's take a break! **] 
 
<div class="countdown" id="timer_540e64b3" data-update-every="1" data-play-sound="true" tabindex="0" style="right:0;bottom:0;margin:5%;font-size:8em;position: relative; width: min-content;">
<div class="countdown-controls"><button class="countdown-bump-down">−</button><button class="countdown-bump-up">+</button></div>
<code class="countdown-time"><span class="countdown-digits minutes">05</span><span class="countdown-digits colon">:</span><span class="countdown-digits seconds">00</span></code>
</div>

__Next session__:

[07-Anonymising](07-Anonymising.html) In order to provide other people to work with the data, a first level of data anonymisation should be implemented